System and method for enhancing the visual effect of a video display

ABSTRACT

A system and method for enhancing the visual effect of a video display. Such a system includes a video display that displays moving images and a light source connected to the video display for providing a light effect that softens visual boundaries of the video display and enhances overall visual effect, wherein the light source illuminates one or more surfaces adjacent to the video display.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of Ser. No. 11/187,824, filed Jul.25, 2005, entitled “VIRTUAL STAGING APPARATUS AND METHOD,” which is acontinuation application of Ser. No. 10/096,310, filed Mar. 13, 2002,entitled “VIRTUAL STAGING APPARATUS AND METHOD,” which is a continuationapplication of Ser. No. 09/359,796, filed Jul. 26, 1999, entitled“VIRTUAL STAGING APPARATUS AND METHOD,” now U.S. Pat. No. 6,386,985,issued May 14, 2002, the subject matters of which are hereinincorporated by reference in their entirety.

BACKGROUND

The image of live actors interacting with animated or pre-recordedimages and sound is commonplace. In film or television, the coordinationbetween the various images and sounds is largely achieved in the editingroom, or as the result of repeated rehearsal until the timing of thecombined performance is exact. In such cases, the combined image isrecorded and, therefore, each viewing is identical. With pre-recordedimages, unfortunately, there is no mechanism to adjust the flow of thepresentation to allow for variations in the performance or the audienceresponse. All aspects of the performance are predetermined and thereforethe presentation is passive to the audience's reaction.

The use of pre-recorded audio and visual material in live performance isalso commonplace in theatre, music, and business presentations. Again,however, there are significant limitations with the ability tomanipulate the pre-recorded material and electronic media incoordination with the live performance. Visuals effects, sounds andother media can be linked together, but such synchronizationcapabilities are currently limited and typically include significantmanual intervention. For example, a technician may, in response to someaspect of a live performance, manually trigger video, sound and lightingchanges together. Once triggered, the various effects typically run fora fixed time. It is therefore necessary for the performers to haverehearsed their timing exactly and for the related media and controlsignals to have been pre-recorded and matched together exactly. Forinstance, once an actor has embarked on a dialogue in suchcircumstances, there is no mechanism to adjust the pace and/or timing ofother media employed concurrently in response to variations in theactor's performance or the audience's response. Accordingly, it isbelieved that the use of pre-recorded media in live performance has beenrestricted to highly rehearsed and short sequences.

U.S. Pat. No. 5,790,124 (the '124 patent) discloses a system forallowing a live performer to interact with a background display screenthat is driven by real-time generated, textured mapped graphics. Bytracking the position of the performer, the system of the '124 patentcan control and change the production of visual images as the performermoves on the stage to give the illusion to an audience that theperformer is moving about a virtual scene. However, the system of the'124 patent is still sensitive to unexpected interruptions, disruptions,and other events that affect the timing of the performance.

SUMMARY

An embodiment described herein provides a system and method forenhancing the visual effect of a video display. Such a system includes avideo display that displays moving images and a light source connectedto the video display for providing a light effect that softens visualboundaries of the video display and enhances overall visual effect,wherein the light source illuminates one or more surfaces adjacent tothe video display.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 a is a diagram of an apparatus according to a preferredembodiment of the present invention.

FIGS. 1 b-1 d are front views of a portion of the apparatus depicted inFIG. 1 a.

FIG. 2 is a block diagram of virtual staging software system objects andmodules according to a preferred embodiment of the present invention.

FIG. 3 is an illustration of a performance director console screenaccording to a preferred embodiment of the present invention.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

A. Overview

Before describing a preferred embodiment with reference to the figures,a brief overview of the invention and its advantages is presented.

The present invention preferably provides for “virtual staging,” which,as used herein, involves creating a virtual experience that combines aperformance or presentation (both of which are hereinafter in thedefinition of the term “performance”) and various electronic and/ormechanical media to create the visual and/or audio illusion of an eventin which a live mass audience (i.e., more than an individual or a small,closely tied group) perceives the sensation of observing or sharing aspace with the media and possibly performers. The “space” is thesetting, surroundings, environment, or context of the performance and ispreferably a simulated three-dimensional space generated by a real-timecomputer graphics engine. In order to create the desired virtual stagingeffect, electronic and/or mechanical media are manipulated in closesynchronization to the performance and the complex confluence between aperformance and multiple electronic and/or mechanical media are managed.Exemplary electronic and/or mechanical media include computercontrollable sensory stimuli devices, such as, for example, videosurfaces, sound systems, lighting systems, and moveable platforms withina live performing area. In a preferred embodiment, a large projectionvideo surface, driven by a real-time three-dimensional graphics engine,is positioned on a stage behind real performers so as to provide avirtual backdrop set through which the performance may navigate.

The present invention can be used to stage many types of performances,including, for example and not by way of limitation, theatricalperformances, concerts, business presentations, and educationalpresentations. A performance, whether live or a playback of apre-recorded performance, proceeds to some extent according to aschedule, but the present invention provides the flexibility andresponsiveness to gracefully handle actions not normal to the schedule.Such non-normal actions include, for example, unscheduled actions, theinevitable and unpredictable variations or disruptions in a performanceitself, the inevitable and unpredictable variations in audiencereaction, or external disruptions (and/or variations in the duration ofthe foregoing). The present invention is capable of compensating fornon-normal actions by manipulating the speed, sequence and/or othertiming aspects of multiple media.

A computer system according to the present invention comprises a scriptof a performance, a control program, and one or more controls connectedto devices whereby sensory stimuli associated with the performance canbe commanded. The script is a technical narrative of the events andactivities that make up the performance, including interdependenciesbetween different events and activities. The script may be organized asseveral simultaneous threads wherein each thread is one particularsequence of events. Exemplary threads are performer dialogues, musicalscores, sound effects, lighting changes, platform motions, videoexcerpts and combinations of the foregoing. The control program executesor “steps through” the script by interpreting each elemental event in asequence, effecting and/or awaiting its occurrence, and managinginterdependencies where necessary, including inherently unpredictableinterdependencies arising from audience reaction or performer/presentervariation. The control program utilizes a timing framework to providefor synchronization and speed variations. The computer system of thepresent invention may also provide interfaces connected to the scriptand/or the control program whereby a performance director and/or theaudience can influence the performance, such as, by adjusting the timingof events and choosing storylines.

As contrasted with conventional technology, the present invention offersseveral significant advantages in terms of the end product performance.Most notably, highly sophisticated combinations of real-lifeperformance, staging effects, sound and visual effects that have beenpossible only using editing techniques can now be performed live withthe present invention. Although some of these effects technically couldbe performed without the invention described herein, the practicallimitations are severe and preclude all but simple combinations.Furthermore, virtual staging offers other significant improvements tothe quality of the theatrical performance, including: (1) improvedsynchronization of effects from many disparate technologies; and (2)realistic navigation through three-dimensional scenes.

From a productions point of view, the present invention offersadvantages in terms of centralization, repeatability, scalability, andportability. First, aspects of the performance, such as music score,dialogue script, acting prompts, staging and lighting directions, andeven control signals to hydraulic staging devices, can be displayedand/or scheduled in close synchronization with each other using acentral facility. Second, once a virtual staging script has beendefined, device control signals and the recorded media created andstored on the computer, the performance can be repeated with ease.Third, the performance is scalable. For example, the video and soundsignals can readily be scaled to different stage and auditoriumconfigurations. If automated lighting rigs are not available, then thesecontrol signals can be used instead to generate manual cues for lightingengineers. The musical score can be replaced with pre-recorded music iflive musicians are not available. In addition, the virtual stagingsoftware can be designed to accept dimensional information about theauditorium and make necessary adjustments to the sound, video andlighting controls automatically, in effect auto-configuring the virtualstage for the location. Fourth, the performance is portable, because avirtual stage does not require the complex mechanics of a traditionaltheatrical set with wings and flys and physical scenery. Instead,material can be stored and copied to new locations quickly, where astandard rig of sound, projection, lighting and stage equipment has beenset up. Also the stage itself can be portable, allowing for performanceson the road. The scalability and portability of the present inventionmake it possible to use the present invention in a more limitedenvironment, such as a home or school, for example.

The underlying technology of virtual staging provides a number ofrelated benefits which include: (1) the ability to create a library ofstatic and moving scenes/backdrops, which may include detailed lightingdirections and sounds effects, such that new scenery can be quicklygenerated and past investments re-used; (2) easy integration of othertechnologies, such as, for example, real-time puppetry where an operatormanipulates a suit or other device and a computer generates a movingcartoon character, especially when open systems standards are utilized;(3) improved stage/audience area ratio, because extensive wing space andspace above the stage are not required, all sets being mostlyelectronically generated and stored, making it possible to utilizevirtual staging in a more regular shaped cinema space; (4) the physicalvolume of the staging equipment may be significantly less thantraditional staging structures, making a virtual staging basedperformance easier to transport between locations (and as noted above,less demanding of space requirements); (5) enhanced safety, as there areminimal moving parts to the set; (6) change to any aspect of theperformance can be more easily implemented and documentation morereadily maintained, because of computerization; and (7) the technologybehind virtual staging can be utilized to support other activities(e.g., business presentations, video games, karaoke, or karaoke theater)that exploit the same location and the overall space could be quicklyreverted to cinema use. For example, the present invention could beutilized as a “karaoke theater,” where monitors are provided to displaycues and lines for participants, while the system generates projectedscenery, music and sounds as necessary to complete the performance.Additional advantages of virtual staging are readily apparent from thedetailed description below.

B. Physical Components

FIG. 1 a is a diagram of a preferred apparatus 100 according to thepresent invention. The primary components of the apparatus 100 are astage 105 and a computer system 110. FIG. 1 b shows a front perspectiveof the stage 105 bare. FIGS. 1 c and 1 d show a front perspective of thestage 105 in use. Positioned to the front side of the stage 105 may bean audience area 115. The stage 105 comprises an performing area 120,which includes several hydraulic platforms 125. Behind the performingarea 120 is a video surface 130, which is flanked on the sides by atranslucent surrounding 135. The apparatus 100 also includes a soundsystem, including speakers 140 and an amplifier 145, and a lightingsystem, including lights 148. The components of the apparatus 100 aredescribed in greater detail below.

The performing area 120 is made up of one or more generally horizontalsurfaces, some or all of which may be moveable, such as the moveableplatforms 125. The computer system 110 generates control outputs tomechanical devices that move the platforms 125 in synchronization withother media to create the combined effect of a virtual stage. Theplatforms 125 may be equipped to move laterally, move vertically and/orrotate. These movements can be synchronized with the perceived angle ofobservation of projected images on the video surface 130, directionalsound effects, adjustments to physical lighting and other media.

The video surface 130 is preferably a large projection surface. Theimage is preferably generated using rear projection technology to avoidactors and physical objects casting shadows on the surface. Suitablemechanisms are video wall technologies and computerized image rearprojectors. The video surface 130 may be curved in the horizontal andvertical planes (most likely a shallow parabolic curvature) to enhanceviewing from seating positions not directly in front of the stage 105.The performing area 120 may also be domed, sloped or curved to betteraccommodate more elevated audience positions and/or to mask thedistortions of perspective arising from the two dimensional projections,the performance area 120 may be irregular or curved. The floor of theperformance area 120 and the platforms 125 also may be transparent, orthin in construction such that the audience can see through to any videosurface below when in an elevated position. Furthermore the videosurface 130 itself need not be rectangular in overall shape, nor limitedto being an individual surface, depending on the staging effectrequired.

It may be desirable to utilize a mounting framework for the videosurface 130 to support the weight of the individual projection devicesand allow movements of the video surface 130 during the performance(e.g., to change the configuration of the video surface 130). A mountingframework preferably allows for quick replacement of individualprojection devices that might become defective.

The gap between the outside edge of the video surface 130 and theboundaries of the staging area may be filled with a textured,translucent or semi-translucent surface 135. This can be illuminatedwith sympathetic lighting colors to visually smooth the edge of thevideo surface 130 and complete the overall visual effect. Multiplediffuse RGB rear projection devices, or some equivalent means, can thenbe used, under the control of the computer system 110, to illuminate thesurface 135 with colors that match the corresponding adjacent edgeportion of the projected video image. The effect is to soften the visualboundary of the projected image on the video surface 130, which mightotherwise be a distraction to the audience and degrade the virtualstaging effect.

To ensure that action on the performing area 120 is merged with theprojected image, a number of devices may be employed. First, lightingfrom above, below and/or the side of the performing area 120 can be usedto balance the relative intensity of the projected image and thephysical objects and actors on the stage. This lighting is preferablypositioned such that it does not interfere with the projected image, asexplained in greater detail below. Second, fluorescent make-up and/orpaint, combined with ultraviolet (UV) lighting can be used to increasethe visual intensity of actors and physical objects. In this regard,note that some objects may be implemented as a combination of physicalobjects and projected image. For example, a chair could have a realbottom seat area and a projected back. Third, the platforms 125 can bemoved so that the actual and apparent movements of physical objects onthe stage surface and the projected image respectively are closelysynchronized. For example, an actor may be raised from below the stageas the projected image itself is scrolled up, giving the audience theimpression that they are, correspondingly, being lowered into the scene.

The apparatus 100 preferably includes a three dimensional wrap-arounddirectional sound system to permit movements on the projected image tobe synchronized with the apparent position of the source of a sound.Directional sound can be accomplished by adjusting relative volumes andpossibly introducing slight delays in broadcasting any particular soundeffect from speakers 140 positioned around the auditorium to give theimpression of sound sources being in or moving through any positionsurrounding the audience, in coordination with projected images. Forexample, a passing car may start as a sound located behind the audiencearea 115, move through the audience area 115 and apparently into thedistance in coordination with the visual image of a car coming into viewin front of the audience area 115 and driving off, as if through theback wall of the stage. In a typical configuration, sound effects wouldbe held in a pre-recorded format, such as MIDI control files, on thecomputer system 110 and referenced by a virtual staging script. Thedirectional effect of the sound can be achieved by adjusting therelative volumes and sound mixes between the speakers 140 positionedsurrounding the audience. Audio outputs can be individual feeds to eachspeaker, such that the computer system 110 determines the relativevolumes needed to establish the observed direction and movement of asound source. Alternatively, a single sound output and control signalcan be generated and fed to a specialist broadcast device. In such aconfiguration, the specialist broadcast device would be capable ofadjusting the relative volumes to the array of speakers 140 based on thedirectional parameters that would be passed to it from the computersystem 110. Alternatively, sound delivery may be by means of headphonesprovided at each seat of the audience area 115.

A physical lighting system, including for example the spotlights 148, isused to balance the visual intensity between performers on theperforming area 120 and images projected behind them. Some parts ofphysical acting props or the acting surface itself may often need to becast in shadow to sustain the virtual staging effect. For example, if amotorbike facing the audience is used as a prop, with the projectedimage and spatial sound being used to create effect of driving along astreet, the lower half of the bike might be in shadow and the rider bemore intensely illuminated. This lighting arrangement would encouragethe observer to match up the actor with the sound and moving image,while ignoring the static surface below. Additional physical lightingeffects can coordinate the generation of light and shadow in thecomputer generated projected image with the incidence of physicallighting. As with all activities, this more elaborate confluence ofmedia is accomplished by the computer system 110 and referenced by thevirtual staging script.

The description above covers the four main components of the virtualstage, namely (1) the projected images, (2) the directional sound, (3)the moveable performance surfaces and (4) physical lighting controls.However, the present invention is capable of managing other devices andmedia in addition to these, as might be required to enhance the virtualstage for a specific performance. This could include pyrotechnics,flying wire mounted objects, scanning lasers projected over theaudience, additional projection surfaces positioned around theauditorium, low frequency sound systems to enhance the sense of movement(e.g. Sensurround (™)), wind generators, smoke emitters, mist emitters,odor generators, and seat motion devices. Devices such as windgenerators (e.g. fans), smoke emitters, mist emitters, odor generators,and seat motion devices may be provided from one or a few global sourcesor may be provided discretely at each seat in the audience area 115.Furthermore, it is even possible to replace the video surface 130 andthe sound system, including the speakers 140 and the amplifier 145, allof which are global scale sensory stimuli devices, with individualizedsensory stimuli devices, such as virtual reality goggles and headphones,respectively.

The complete effect of a virtual staging experience is observable by theaudience within an “observation cone.” The observation cone is a threedimensional space emanating out from the stage 105 and defined to be thespace within which all the media and observed actions combine withinallowed tolerances to sustain the impression of the virtual stage.Members of the audience positioned within the observation cone will allobserve slightly different images of the virtual stage due to theirdifferent relative positions, but the combined effect of the virtualstage will be consistent. For example, as an actor stands in front of aprojected three dimensional image of a wood, a member of the audienceseated to one side of the observation cone will perceive that the actoris standing in a different relative position to the trees in the wood,as compared to a member of the audience seated on the opposite side.Nonetheless, both will see an actor standing in a wood. If the platform125 of the performing area 120 is rotated in synchronization with theobserved perspective of the projected image of the wood, both members ofthe audience will experience the effect that they are rotating theirposition around the actor, as a cameraman might when making a film.

With two dimensional visual projections of three dimensional views,there is a limit to the angle of observation for which the illusion ofchanging perspectives can be sustained. This limitation defines theobservation cone in virtual staging. Outside this angle, the variationsin perspective become too great for the observer to perceive the effect,as the individual components become distinguishable. The observationcone is also limited by the projected image, as described above; thedirectional capabilities of sound in the auditorium (although theobserved direction of a sound effect is far less precise for members ofthe audience); and the physical lighting system.

C. Human Components and Interfaces

The three primary human components associated with the virtual stagingapparatus 100 are performers, the audience, and a performance conductor.These people and their interfaces to the virtual staging apparatus 100are next described.

The performers act out their roles on the performing area 120. They arenot restricted to this area other than as necessary to sustain thevirtual staging effect. An actor may move into the audience area 115 fordramatic effect, for example, at which time the virtual staging effectbecomes secondary to the performance. The performers may respond tosound and visual cues or provide cues to the performance conductor totrigger the next sequence of actions set out in the virtual stagingscript. It is possible that a performer may be provided with a hand helddevice on stage to be able to undertake the performance conductor rolefor part or all of the performance. It is also possible to includemovement sensors and/or position indicators attached to the performersor the performing area 120 to generate prompts or inputs to the computersystem 110 or other devices to trigger responses that further enhancethe overall virtual staging effect. Performers are not required,however. The virtual staging experience can be accomplished entirely onthe basis of electronic and/or mechanical media.

The audience area 115 is preferably positioned within the observationcone, as described above. The audience perceives the virtual stagingexperience and reacts to it. In the simplest form, the audience'sinfluence over the performance is limited to the variable time taken toreact to the presentation (e.g. applause or laughter). In a moresophisticated implementation, the audience can influence the directionof the performance itself, collectively instructing the performers,performance director or the computer system 110 to take one or anotherpath through the virtual staging script. One mechanism for collectinginput from the audience is input devices, such as voting buttons,provided on the audience's seats.

The performance director/conductor is a role that can be taken by anindividual or shared between several people. A performance conductor mayor may not be a performer. The performance conductor observes theperformance and interfaces with the computer system 110 to control andmanage the progress through the virtual staging script, responding tothe variations of timing in performance and/or audience response. Theperformance may be presented as a single story line, a story line withmany allowed paths, or even a number of building blocks that can becombined together in different sequences for each individualperformance. The performance conductor is provided with the necessarycontrols to adjust the sequence of activities where there are allowedchanges in the story line. The audience may assume part or all of theperformance director's role. In some cases the performance conductor maybe absent or optional. In such cases, the script is completelypredetermined, and the computer system 110 itself schedules all events,cueing performers and devices as necessary to run through the script inan automated fashion without human intervention or perhaps with onlyminimal human oversight.

In addition to the three primary human components just described, otherhuman components are possible, including musicians, engineers, andothers. For example, for full flexibility, accompanying music is ideallyperformed live. In this configuration, the computer system 110 stores amusical score and presents it at the appropriate time on one or moredisplay consoles visible to the musicians. In addition to the musicalscore, the displays can provide cue and beat information to enable themusicians to synchronize to the overall virtual performance

D. Computer Hardware

The computer system 110 produces the virtual staging experience bycontrolling physical components and interfacing with human components.In the preferred embodiment of FIG. 1, the computer system 110 comprisesa local area network (LAN) that links a central controller 150 tovarious devices, files, and consoles, such as an imageprocessor/framestore 155, performance director console(s) 160, performerconsole(s) 162, engineer console(s) 164, other console(s) 166,image/audio files 170, sound mixing deck 175, stage device controllers180, and specialist equipment, such as specialist sound/image device185. One skilled in the art will realize that other connection networksbesides a LAN are equally feasible.

Virtual staging software, as described in the next section, can be madeto run on any hardware platform with adequate capacity and capability tosupport the real-time generation of video images, sound and devicecontrol signals. The computer system 110 preferably has recovery andback-up capabilities as well as configured redundancy to protect againstsingle and multiple failures of major technical components. Theconfiguration is preferably “open” to allow for the integration ofadditional LAN based servers and facilities that might be required(e.g., back-up/recovery devices, print servers, or even devices tosupport unrelated uses.)

The computer system 110 includes an image processor/framestore 155 sothat a single projected image can be routed to the video projectionscreen 130. In the case of video wall technology, one or more videoimage framestores 155 split the overall image between the individualprojection devices as desired. For rear projection devices, the videoimage framestore 155 is a projector control unit that synchronizes thedisplays where more than one projector is used. As most commercialframestores are designed for rectangular projection surfaces, it may bedesirable to “square off” irregular shapes with dummy screens (i.e.stubbing devices that will absorb and suppress the video signal for thecorner areas where no actual projection device is needed, while allowingthe framestore to function normally).

The computer system 110 is capable of receiving (via the LAN or othercommunications facility) and merging additional video signals with itsinternally generated images, if desired. Such an interface can be usedto incorporate real-time interactive cartoon characters, for example topermit an electronic puppeteer to manage the projected image of acomputer generated character on a stand-alone device and output theimage via the central controller 150. Alternately, an interface may besupported by routing directly to the projection controller or framestore155.

The sequence of projected images, sound and stage movements are managedusing the central controller 150, which permits the performancedirector(s) to manage the scheduling and sequencing of activity,coordinating with the actors, musicians (if live music is being used),lighting and staging engineers, etc. The main output from the centralcontroller 150 is one or more real-time video/image feeds, as requiredby the image processor/framestore(s) 155, and sound signals. Additionaloutputs from the central controller 150 may be electronic messages todevice controllers (e.g., sound effect facilities, pre-recorded sounddevices, lighting controllers, stage hydraulic devices). In addition,the system can support user configurable monitor consoles 160, 162, 164and 166, which the performance director(s), actors, musicians, lightingand staging engineers and others can observe to coordinate their aspectsof the performance.

E. Software Design

FIG. 2 is a block diagram of a virtual staging software system 200 thatexecutes on the computer system 110. At the heart of the software systemis a script 205, a control program 210, and a timing framework 215. Thescript 205 lays out the schedule of events, as described below. Thecontrol program 210 executes the script 205 by interfacing to variousdevice/system controls, as well as user consoles. Specifically, FIG. 2illustrates video processor 155, sound system control 220, lightingsystem control 225 and various consoles 162-168 interfaced to thecontrol program 210. The various device/system control interfaces aswell as user consoles 155, 220-235, and 162-268 are utilized to augmenta live performance by providing sensual stimuli to the audience directlyor indirectly. Other important components of the virtual stagingsoftware system 200 are the database 170, and interfaces 240,245 to theperformance director and audience respectively. The performance directorinterface 240 and the audience interface 245 are connected to eachother, the script 205, and the timing framework 215. Although objectsand modules in FIG. 2 are conventionally implemented in software, oneskilled in the art would realize that implementations in hardware,firmware, and combinations of hardware, firmware and software areequally feasible.

The objects and modules in FIG. 2 (e.g., the script 205 and the database170) may be embodied on any computer readable medium, including storagedevices and signals, whether modulated using a carrier or not, that thecomputer system 110 can be configured to access. Typical computerreadable storage devices include conventional computer system RAM(random access memory), ROM (read only memory), EPROM (erasable,programmable ROM), EEPROM (electrically erasable, programmable ROM), andmagnetic or optical disks or tapes. Such computer readable storagedevices may be removable or permanently attached to the computer system110. Exemplary computer readable signals include signals downloadedthrough the Internet or other modem connected data source.

To the extent possible, it is desirable that the objects and modules ofthe virtual staging computer system 200 conform to accepted orstandardized formats and protocols. For instance, computer images andsound items stored on the database 170 preferably use industry standardfile formats. Object oriented (OO) analysis and design methods are themost suitable for a custom software development of the overall systemcontrol facilities, as each segment of sound, image, musical score orany other item of the performance is best treated as an instance of ageneric OO class. Suitable development languages for the control programinclude JAVA or C++, or some combination of the two. Any custom softwarepreferably should conform closely to emerging OO and Open Systemsstandards to ensure easy integration with commercial software andhardware. For example, the drawing environment, music scoring, digitalimage file management and device drivers are preferably integratedcommercial products. Commercial product software development, releasecontrol and production management standards are preferably enforced forall custom software implementations.

In a preferred embodiment, the virtual staging software system 200represents a performance as the script 205. The script 205 is thetechnical narrative of the show/presentation and enables the performancedirector to track the position in the performance and manually triggerevents. The script 205 may contain a number of threads 1 . . . N ofactivity that are executed alongside each other and which may havecomplicated interdependencies. For example, a set of threads may includethe musical score for an instrument, the spoken lines for a character,the sequence of set images for the video surface 130, lightinginstructions, or a series of sound effects. Tight coordination betweendifferent threads is desirable, e.g., it is better if all the musiciansplay in time and the actors speak their lines in the intended sequence.The script 205 may comprise event sequences that are predetermined,assembled on the fly slightly in advance of their execution, or as ismost likely, some combination of the two.

The script 205 is executed by the control program 210 using the timingframework 215 and associated controls under the direction and/oroversight of the performance director. The script preferably records thelinks between threads of activity that need to be coordinated, and theperformance director uses a suite of controls to manage the exact timingand rate of execution of the script to match variations in the liveperformance. The execution of the performance is in two distinct modes.One mode is event driven progress in which the performance moves forwardas the performance director checks off each event as it is completed, oras the control program 210 receives electronic notification ofcompletion from a device controller or equivalent. A second mode ofoperation is a time based or automated sequencing, where scheduledevents are displayed and/or triggered (where automated device driversare implemented) in real-time. For example, when in this second mode,the music score and staging instructions would scroll automatically. Theperformance director has comprehensive controls to oversee theperformance including the capability to start, stop, jump forward andbackward, speed-up and slow-down the rate of progress in this second,automated mode of operation.

The timing framework 215 may comprise a clock to provide a “heartbeat”for stepping through the script 205. All threads may execute accordingto the same heartbeat clock, or separate heartbeat clocks may beprovided for separate threads. As just mentioned, the performancedirector has the ability to accelerate or decelerate the heartbeat, skipa given number of beats, reset the beat count and perform various othertiming modifications to the timing framework 215. The timing framework215 may also provide timers to, for example, trigger event executionafter a fixed delay or extend a condition for a given time.

The script 205 may reference items stored in the database 170. Given themultimedia nature of the items held, an object database or equivalent isbest suited for the implementation. The organization of the database 170is designed to support the quick identification and retrieval of items.A useful aspect of the database organization is item categorization andindex structures. In addition, the database 170 can store a library ofitems from which new sets can be rapidly assembled, support fullinventory management and support for other activities such asdevelopment, testing and release control. The database 170 can alsostore references to items that may be located in remote data-stores ornetworks, including the Internet, such that images and sounds from othersources can be worked into the overall presentation. Typical itemsstored in the database 170 include video images, audio signals, staginginstructions, dialogue script instructions, and music scores, as morefully described in the following paragraphs.

Full video images (or partial if supported by the projection devices)are routed to the video surface 130 via, for example, a projectiondevice. The images may be (1) pre-recorded video clips or (2) computergenerated images, such as images created by a real-timethree-dimensional graphics engine.

In the former case, the video image items preferably have associatedparameters appropriate for its format, including duration for finitelength sequences, repeat cycle time for video loops, display ratecontrol information where the sequence can be run at variable speed (forexample as might be required to coordinate with some physical stagingdevice). In addition, image size and location parameters may beincluded. The performance director, or possibly a dedicated videomanager will use the computer system 110 to sequence the video signals,i.e. initiating the next video sequence, terminating a video loop orchanging the static image displayed.

In the latter case, video image will be generated in the centralcontroller 150 (or linked peripheral devices) from a stored threedimensional space. A good example of this method of image creation isused in video games, where the user navigates the stored threedimensional space using the game controller and the program derives amoving screen image that mimics the movement. For this format of image,an additional facility can be supported where the performance directoror video manager has the ability to manipulate the space represented bythe projected image on the stage, using some convenient peripheralcontrol device, as in a video game. The main function of the videomanager, however, is to use the system to coordinate the video displaywith the acting, triggering screen images and sounds in synchronizationwith movements on stage. This is referred to as “dynamic actorinteraction,” where real movements are enhanced or embellished withmoving images and sound. This combination of reality and imagery is usedto create powerful visual effects.

Audio signals and sound effects are managed in a manner similar to thevideo signals. Where appropriate, sound effects will be combined withvideo images to simplify the control. The video storage format ideallyallows for recording the audio signal with the image, eliminating theneed for a separate sound-signal management facility for many soundeffects. For example, MPEG (moving picture experts group) videocompression standards, or a three-dimensional extension, allow forrecording accompanying audio signals.

Lighting instructions, stage movements and sound instructions also maybe recorded in the database 170 in this manner and referenced by thescript 205. Preferably the virtual staging software 200 integrates astandard word processing product for the generation of these items.

If live music is performed and needs to be coordinated with events in aperformance, then instrument scores can be held in the database 170 anddisplayed in a format linked with staging events and triggers so thatthe musicians can match their timing to the performance. Similarly,where the music needs to be coordinated with staging events, theperformance director can ensure that rate of progress is matched to thespeed with which the music is being played (using the timing framework215 controls listed above) to ensure actions are cued on time.

The users of the computer system 110 include three main types:performance director(s), performers and musicians, and engineers. Theuse of the virtual staging software system 200 from the perspectives ofthese three types of users is now described.

The performance director(s) uses one or more monitors to track allstreams of activity. The performance director is able to dynamicallyconfigure what is displayed on the performer and engineer's consolemonitors (see below); check off events, preferably with a simple mouseinstruction; and, when in an automated mode, manage the rate ofprogress. The primary task of the performance director, or an additionalvideo manager for more complex work, is to trigger the displays insynchronization with the stage acting. The system will provide theability to see and possibly adjust future scheduled images on a localscreen before they are committed to the main display(s). Where theprojected image is derived from a three dimensional space maintained inthe central controller 150 (or linked system), the performance director(or video manager) will also be able to manipulate the apparent positionand movement through the three dimensional space (as perceived by theaudience) using a suitable control.

Monitor consoles can display a music score, dialogue script and stagingdescriptions in a user configurable form for musicians and performers.The display indicates the estimated point at which the overallperformance has reached. In an event mode, the display moves forward asthe event director checks items off. In an automatic mode, the displayscrolls forward at a rate managed by the performance director.Information can scroll horizontally or vertically depending on the mostconvenient representation (i.e. music score may scroll horizontally anddialogue script vertically, in different screen windows). Each user hasthe ability to move forward and backward in the schedule (“look ahead”),and revert to the current position independently of all other users.

Monitor consoles 164 for sound, light and stage engineers are usedsimilarly to those for performers, but display a different profile ofinformation suited to a specific engineer's requirements, e.g. lightinginstructions along side dialogue script. Automatically triggered eventscan be highlighted on the monitors, e.g. the timing of an automatedpyrotechnic would be clearly displayed. The user preferably has anoverride capability for certain automated events for safety purposes.Such a feature could be implemented as part of a configurable userprofile.

The software system 200 supports online and offline facilities underconfigurable user profiles. The online facilities will first bedescribed. Display configuration (i.e., select items and presentationformats for the console monitor) is preferably supported online.Performance schedule amendments and modifications, supported by a suiteof online editing facilities, are supported online. In an event drivenmode, supported scheduling operations include moving the “currentposition” forward one step, backward one step, jumping forward, jumpingbackward, repeating one or more events, awaiting external trigger. In anautomated mode, supported scheduling operations include speeding up therate of progress, slowing down the rate of progress, starting from aposition in the schedule, stopping at a position in the schedule,looping from one position to another, and skipping over a portion of theschedule. Another online facility is moving the displayed portion of theschedule forward or backward (without changing the “current position”setting) to look ahead to what is coming. Other important onlinefacilities are manually overriding an automated event (e.g. suppress asignal to a staging device such as a pyrotechnic); manually overridingan interrelated dependency; and broadcasting a message or sending amessage to a specific user.

Various facilities are also provided offline. First, setting up defaultuser profiles (e.g., monitor layout and permitted functions) ispreferably an offline facility. Other offline facilities are storing andlabeling a performance item in the database; defining a performanceschedule (sequence and timing of items) using editing utilities;recording different item types in suitable formats (e.g. score anddialogue script), including the links between them, in a meaningfulmanner (e.g. a lighting change during a song); creating dependencies(e.g., one event may be defined to wait on the completion of another, orautomatically trigger another) where the central unit has an automatedinterface to the device that executes the item, such as a sound card,which could notify the central unit when it has completed playing asound item. Another offline facility is launching an item creationprogram, especially if the primary virtual staging software 200 will notitself support the creation of items other than the performance scheduleitself. Preferably, items such as a dialogue script and staginginstructions can be created using simple word processing utilities.Also, for example, video and audio items can be created using commercialproducts that can either interface directly with the database 170 (e.g.a scanner writing a fixed image to a predetermined file space) or whichcan run on the central controller 150 or a specialist server on the LAN.These programs can be called from the central program as utilitiesduring the development of items (e.g. the drawing of a new image) orduring the performance (e.g. to integrate a sound effect device).Finally, system controls and administrative facilities are preferablyprovided off-line through a separate series of screen based facilitieson an operations console.

The primary non-user interfaces to the computer system 110 are to devicedrivers and peripheral units. Administrative and security/backup relatedinterfaces also can be supported. The main automated outputs are thevideo and the sound effect feeds, to the projection device and soundboard respectively. Live sound is mixed with the audio feed at aseparate sound desk, the audio feed being treated basically as anadditional input. The audio feed can carry positional information tosupport the directional/spatial sound requirements. Other interfaces caninclude triggers to staging devices and pyrotechnics as desired andavailable, and support for additional sound and image inputs from otherstandalone devices (such as the real-time cartoon character devicedescribed earlier, other video sources or sound effect machines).

F. Example Scene

The following description outlines how a performance director would usevirtual staging to manage a typical sequence of related events in aperformance. In the example the action of the sequence as perceived bythe audience is first described. Then, a screen appearing on aperformance director console is described. Finally, the action is crossreferenced to the details on the example screen, making specificreference to virtual staging controls.

1. Action Sequence

Three actors stand looking out over the river Styx, their backs to theaudience with the river projected as an image behind them on the stage.One character plays Moses, who turns away from the river to face theaudience and starts to assemble his staff in preparation for parting thewaters. As he is so preoccupied, one of the other actors stoops down andappears to skim a flat stone out over the (projected image of the)river. Instead of skipping across the water as might be expected, thewater mysteriously forms a sinister mouth and swallows the stone as itnears its surface. Moments later, unaware of what his fellow travelershave just observed, Moses turns back to the river and raises his staff.As he chants, the river is torn in two in front of the actors andaudience.

Quickly the three travelers climb onto their bikes and start to pedaltowards and seemingly into the channel that has opened up in the river.As they motion as if to pedal away from the audience and into the river,the perceived observation position of the projected image and relatedsound effects are changed, to create the impression that the audience isfollowing behind the actors, into the watery channel. As the actors andaudience “move” forward, the apparent source of sounds of the river aresimultaneously moved backwards through the auditorium, embellishing theimpression of motion across/through the river.

Part way across, the introductory music to a song starts. At this timethe perceived angle of view of the projected image, directions of soundsand the acting surface itself are all rotated in coordination. Theeffect experienced by the audience is as if they have orbited 180degrees around the actors, moving from a position following them, to onewhere they now look at them, pedaling towards the audience. With theactors now pedaling forwards at the audience and the image and sounds ofthe river passing off into the distance behind them, the actors sing thenext song. By the conclusion of the song, they reach the far bank of theriver, as the river channel closes behind them.

2. The Performance Director Console

The performance director manages the execution of the electronic scripttypically by triggering a coordinated sequence of control signals, andpossibly adjusting the rate of execution of any one (combined) sequence.In both instances, the performance director progresses and adjusts theexecution of the script in response to the live performance and/or theaudience reaction. A console displays the various threads of activity ina visual format (e.g. actor's dialogue, musician's score, displayedvideo image or graphic engine coordinate controls, sound effects,lighting adjustments and physical stage movements). The electronicscript maintains timing and synchronization links between theseactivities as necessary to maintain the virtual staging effect, whileproviding the performance director with the flexibility to pause,prompt, accelerate or slow down overall execution as may be needed toadjust to the spontaneity of any one performance.

FIG. 3 shows an exemplary screen 300 of a performance director console.Only the main controls are shown on this simplified version. On a fullscale screen many more threads and a longer sequence of activity wouldbe visible in a main window 305. In addition, a more comprehensive andsophisticated array of editing and manipulative controls would beavailable. The purpose of FIG. 3 is to illustrate the nature of theon-line controls that will be available to the performance director tomanage the performance. The main features of the screen 300 will next bedescribed.

The main window 305 is the large area in the center of the screen and isin fact two separate windows, which can be independently scrolled/pagedvertically and if appropriate horizontally, (as implied by the standardformat control buttons shown). The left window of the main window 305 isa dialogue script view 310 and shows the dialogue script and lyrics,with markers 315 positioned within the dialogue as a visualreference/prompt for the performance director. The right window of themain window 305 is a thread view 320 and shows a selected combination ofthreads that the performance director uses to monitor and trigger therange of effects in synchronization with the performance. Thread labels,such as the thread label 322, at the top of the thread view 320 indicatethe identity of each thread, which progress vertically downward. Timeprogresses vertically downwards on the thread view 320, corresponding tothe script being read down the dialogue script view 310.

Between the dialogue script view 310 and thread view 320 are durationindicators 325. Each vertical block of the duration indicators 325corresponds to a short time interval (e.g. 15 seconds), providing arough guide as to the typical timing of a performance. The verticalblocks might be of different sizes as actors are unlikely to speak at aconstant rate of words or lines per minute. The duration indicators 325are merely a visual aid to enable the performance director to match thedialogue in the dialogue script view 310 to the events scheduled on thethreads in the thread view 320, by providing a common vertical scale. Analternate screen format might use a standard time interval vertically,in which case the text in the script view 310 would need to bereformatted (e.g. variable line spacing) to match to it. A horizontalindicator, such as the current position indicator 330, shows the presentpoint of the performance. In the example it can be seen that theperformance director has scrolled the dialogue script view 310 downslightly, perhaps to look ahead at the dialogue.

The main window (both halves) scrolls down as determined by scrollcontrols 335. The system defaults to scrolling the window at apredetermined rate (based on a typical performance). The performancedirector can then adjust the default rate, speeding up, slowing down orpausing the scrolling in response to the specific performance. Analternative mode of display is to simply step the main window 305forward as the performance director triggers specific events.

The performance director is able to dynamically select which threads areshown in the thread view 320. The thread view 320 window summarizes howthreads (such as video sequences, lighting adjustments, sounds effectsand stage hydraulic controls) are synchronized to the performance. Anevent in a thread could be the initiation of an action, such as startinga video sequence, or could be a state change, such as a lightingadjustment. An event is indicated as a circle on the vertical line ofany visible thread. Key events marked with cross arrows, are events thatwhen triggered, cause the automatic initiation of one or more otherevents in other threads. This synchronization is shown in the diagramwith a dark connecting line between the key event and its triggeredsecondary events. For example, a key event marker 340 (labeled “2” inthis case) is linked to the secondary event marker 345.

An event will typically have a duration, such as the playing time of avideo sequence, or the time taken to rotate a stage turntable. Theduration of an event is indicated by the vertical length of an eventdescriptor—the rectangular box extending below the circle event markerin the diagram—such as the event descriptor box 350. If an eventdescriptor box has a dotted outline, this shows that its duration isunbounded, i.e. it can continue indefinitely and need not besynchronized to any other activity, such as a background sound effect ora lighting configuration. Such an event would typically be terminatedwhenever the next event on that thread is triggered.

Where one or more events need to be executed in lock-stepsynchronization at a matched rate, this is indicated with a shadedbackground in the diagram, such as synchronization indicator 355. Forexample, the execution rates of video, sound and stage turntableposition adjustments may need to be coordinated to achieve a suitablevirtual staging effect as the auditorium is “rotated”.

In typical operation, the performance director will initiate one keyevent after another, in time with the live performance, following thesequence in the electronic script, as shown pictorially on the screen300. Some visual prompt may be provided, e.g. the next key eventflashing on the screen, at which point the performance director canclick on the event trigger button 360 at the exact time to match thelive performance. Alternatively, key events can be triggered by thesystem automatically, in which case the performance director takes moreof a passive role. The subsequent rate of execution of the threads canthen be adjusted if necessary against the default rate using ratecontrols 365 to speed up, slow down, pause or reset to the default rateas necessary. The system responds, for example, by increasing the framerate of a video sequence, rotating a stage turntable slightly faster,and increasing the beat rate for live musicians.

Two further pieces of information are available to the performancedirector. When clicking on an event marker, such as the key event marker340, for example, a small window (not shown) opens with a briefnarrative description of the event. For more detail, the performancedirector can point to the event descriptor box and then drag the cursorinto one of two event viewer windows 370 (shown at the base of thescreen 300). Depending on the nature of the event, additional details ofthe event are displayed, with a suitable selection of display controls.In the example shown, a video image with playback controls is shown onthe left side of the event viewer 370, and some lighting adjustmentdescriptions are shown as a simple text box on the right side of theevent viewer 370.

3. Interaction Between Action and Console

The stage action described above is also reflected in the information onscreen 300 Numbers (1-5) shown in the key event markers correspond tospecific points in the action as now described. Note that for the sakeof this example, the video component is described as if it is generatedusing a three-dimensional real-time graphics engine. Using thistechnology, the virtual staging control framework provides apparentviewpoint position coordinates and motion controls to the engine toenable it to generate the correct perspective and position for theprojected image. The same or similar effect could be achieved usingpre-recorded video loops and sequences. In such an instance, instead ofproviding coordinate and motion instructions, the system would initiatepre-recorded video segments as necessary to match the performance. Ifusing pre-recorded video, loops would be required to project stationeryscenes, and variable rate sequences for moving scenes.

Starting with the three characters standing on the river bank, theperformance director monitors their dialogue, as the scene remainslargely static, with distant sounds of the Underworld surrounding theaudience, according to threads VIDEO 2 and SOUND 1. As Moses turns fromthe river, a second actor approaches a point on the stage, facing thevideo projection screen. The performance director triggers an animatedsequence, which is overlaid on the projected image of the river,according to event no. 1 of the thread VIDEO 1. First, small enough toonly be visible to the actor, a countdown sequence of numbers (4,3,2,1)is projected on the part of the screen where the actor is to pretend tocast the stone. As the video sequence progresses, the system issueslighting adjustment signals to a lighting rig to divert attention awayfrom Moses to the other actor. (Note the lighting thread is not visibleon the screen 300 as shown, though the link between it and the key eventno. 1 of the thread VIDEO 1 is visible as a horizontal dark line.) Atthe exact time the actor swings his arm, a cartoon image of a stoneflies out over the river, only to be gobbled up by the sinister facethat forms in its surface.

In time with the completion of the animated sequence, the system thenissues further lighting adjustments automatically, restoring the initialbalance of lighting. (Again, the specific event on the lighting threadis not shown.) The actors work through further dialogue, as Mosesprepares to part the waters and instructs the others to ready the bikes.The bikes are physical props, mounted on an automated rotating turntableset in the stage surface. As Moses moves to face the projected image ofthe river (facing away from the audience), the performance directorprompts the video sequence of the parting of the water. In time with thevideo image, sounds and lighting changes are made, focusing attention onMoses and triggering suitable surround sound effects, such as risingthunder, according to the threads VIDEO 2 and SOUND 1 at event no. 2. Intime with the dialogue, the video image shows the sky darkening as theriver parts. Coordinated lighting adjustments and low frequency sound isbroadcast into the auditorium in close synchronization with the image.

As the waters part, Moses turns back to the other actors, standing bythe bikes. The performance director and actors wait, in case, as happensoccasionally, the audience reacts with applause to the visual effect.Then the actors climb onto their bikes and get ready to start pedaling.The performance director triggers the next scheduled key event (keyevent no. 3), which starts a moving video sequence as the actors, withthe audience in tow, appears to pedal into the channel in the river.This virtual effect of movement is enhanced as the directionalcapability of the sound system in the auditorium is used insynchronization with the video image to give the impression that thesources of various sounds are themselves moving backwards through theauditorium.

Part way into the river channel, the performance director then triggersan apparent 180 degree rotation of the virtual stage. Synchronizedcontrol signals are sent to the graphics engine which generates outputsto VIDEO 2, to the directional sound effects generator and to therotating stage turntable, as shown at event no. 4. Also at this time, inresponse to a livelier than typical audience response, the performancedirector may decide to speed up their journey across the river, bymaking adjustments to the rate of executing the script, e.g., pressingthe “up” arrow of the rate controls 365 to set the rate to 102%.

Once the actors have turned the full 180 degrees to now be facing theaudience, the Performance director triggers the cue for the musicians atkey event no. 5. Beat and cue information, reflecting the fasterexecution rate selected by the performance director earlier, isdisplayed on the monitors in front of the musicians (as might also betheir musical score). The musicians start to play at the appropriatepoint and rhythm to match the next video sequence of passing through thechannel towards the audience. The system ensures, that even though theperformance director has sped up the execution, the rate of displayingthe video sequence is matched to the adjusted rhythm of the song, suchthat at the end of the song, the actors reach the bank of the river asplanned.

The preceding example is intended to be illustrative, not limiting, ofthe features of the present invention. Those skilled in the art willreadily appreciate that different symbols, designs, and arrangements ofinformation in the screen 300 are possible. Those skilled in the artwill further realize that the various threads of activity, as well asthe linkages between them, may be represented, expressed, stored, andsignified in various forms, and that the same virtual staging scene mayresult from sets of threads that are organized or expressed differently.The terms and descriptions used herein are set forth by way ofillustration only and are not meant as limitations. Those skilled in theart will recognize that numerous variations are possible within thespirit and scope of the invention as defined in the following claims andtheir equivalents.

1. A system for enhancing the visual effect of a video display,comprising: a video display that displays moving images, the videodisplay comprising a video surface having an outside edge; and a lightsource connected to the video display for providing a light effect tovisually smooth the outside edge of the video surface and enhancesoverall visual effect, wherein the light source illuminates one or moresurfaces adjacent to the video display and is capable of providingillumination of different color and intensity, and the light effectincludes colors that are sympathetic to colors displayed in the movingimages on the video display.
 2. The system of claim 1 wherein thesympathetic lighting color is a color that matches a color at a borderportion of the video display.
 3. The system of claim 1 wherein thesympathetic lighting color is a color that is in agreement with a colorat a border portion of the video display.
 4. The system of claim 1,wherein the light source includes a rear projection device, wherein themoving images are generated by the rear projection device.
 5. The systemof claim 1, further comprising a computer system comprising a processorfor controlling the light source.